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Abstract 



This report describes implementation of the DC Century Community Learning 
Center (DC 21®^ CCLC) program during the summer of 2001, with a particular focus on the use 
of computer technology to improve academic achievement. The report is based on direct 
observations of activities, document reviews, interviews with program coordinators and 
faciUtators, and focus groups with student participants. The investigation revealed that the 
technology appears to have been well implemented, with large numbers of well- functioning 
machines in almost all observed classrooms and generally positive reactions from staff and 
students. Student enrollment is somewhat lower than hoped, and observations suggest that the 
use of the Internet, pretests,- and aides could be improved to better support student 
achievement. 
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Introduction 



A great number of education reforms in recent years have been aimed at improving 
students’ academic skills. The lack of concurrent improvement in standardized test scores, both 
nationally and in Washington, D.C.,^ suggests that these reforms may not be working as well as 
hoped Many educators are turning to out-of-school-time activities as a possible area where 
additional gains can be achieved A particularly striking example is the federal government’s 
21 St Century Community Learning Center program, which provides ftinds to schools nationwide 
to improve their out-of-school-time programs. The program was started in 1998 with an annual 
budget of about $40 million. Funding has increased dramatically since then, to around $850 
million for fiscal year 2001 . 

At the same time that interest in out-of-school-time programming is growing, there is 
also increased interest in using technology, especially computers, to improve academic 
achievement. The E-Rate program, which helps fimd Internet access and related equipment and 
services for schools and libraries nationwide, is currently funded at around $2 billion per year 
nationwide and $5-$10 million per year in DC^ and E-Rate is only one of many federal, state, 
and local initiatives designed to increase the use of technology in our schools (Puma, Chaplin, 
and Pape 2000). 

Youth in the District of Columbia face many challenges in terms of both academic 
success and later employment outcomes (Chaplin et al. 1999). DC Public School (DCPS) 



' College Board (2000) and Washington Post ^August 29, 2001 , pp. A 1 and A20. The lack of improvement in 
test scores in Washington, D.C. may be related to an increase in poverty during the 1 990’ s (Rubin, 2002). 
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system staff, like many of their counterparts nationwide, are working hard to improve their out- 
of-school-time programs and bring technology into the educational system. The DC 21st 
Century Community Learning Center program (DC 21st CCLC) is one example of these efforts 
as it provides enhanced out-of- school- time activities for youth in 10 middle and junior high 
schools in Washington, D.C. The program, which ran fiom the fall of 1999 through the summer 
of 2002, was supported in part by a three-year, $4.1 million grant from the U.S. Department of 
Education (DOE). 

DC 21st CCLC focuses on nonacademic out-of- school-time activities,^ but the DOE 
funds are also used to purchase computers and software that can be used during the regular 
school day to improve academic achievement. In this report, we focus on how technology was 
used to improve academic achievement in the DC 2L^ CCLC program during the summer of 
2001 A companion report (Russell et al, 2002) describes non-academic components of the 
program and other related issues. 

Dtiring the summer, DCPS staff operate both a summer school and a nonacademic out- 
of- school-time program. Programs for all 10 DC CCLC schools were in operation from 
June 25 to August 3, 2001, at 8 of the 10 schools where DC CCLC operates during the 



^ DC schools received almost $5 million in Erate funds in 1998, over $50 per student, and most of this was 
for the public school system. DC was awarded another 9 million in 1999. 

^ The DC 2PCCLC program has four components: (1) after-school activities during the school year, (2) a 
summer program, (3) an intergenerational program, and (4) an adult activities program. This report focuses 
on the summer program. Two of our earlier reports covered the first two components of the program 
(Raphael and Chaplin 2000a, b). Currently, there is no plan to evaluate the other two components. 

^ Most of the funding for these nonacademic activities paid for facilitator salaries and came from the 
Temporary Assistance for Needy Families (TANF) block grant rather than DC 2 P CCLC funds. The salary of 
the school-level director of the morning activities was also from the TANF funds. 
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school year. Students from the remaining two schools were allowed to attend at one of the sites 
that was open. 

The summer school was in the morning and the nonacademic program in the afternoon. 
Many students had to attend the summer school, and all students from the participating schools 
(and some others) were allowed to attend either the morning or afternoon sessions. DC 21st 
CCLC funds paid for the school-level coordinator of the out-of-school-time program and for 
equipment, including many of the computers and much of the software lised in the morning. In 
addition, the DC 21st CCLC director hired the facihtators who administered the computer- 
driven academic activities during the morning summer school.^ Thus, DC CCLC played a 
large role in how technology was used to improve academic achievement during the DCPS 
summer school program. 

This report begins with a general description of the technology components of the DC 
21®^ CCLC program, followed by a discussion of its most promising strengths and problematic 
issues. We then make additional observations and summarize the results. 

This report is based on data collected in the following ways: 

• A review of school site proposals. 

• A review of monitoring reports submitted by the DC 21*^ CCLC program director. 

• Observations of 17 academically focused technology activities (morning).^ 

• Interviews with 16 facilitators/instmctors of these activities. 

• Focus groups with students participating in these activities at 7 sites.^ 

• Observations of 3 1 nonacademic activities, including sports, arts, community service, 

and technology (afternoon). 



^ A number of non-computer academic activities take place during the morning summer school program and 
are not covered in this report. 

^ Appendix C contains copies of the protocols and survey instruments we used to collect our data. 

’ Scheduling problems precluded conducting a student focus group at the 8* site. 
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• Interviews with 10 assistant principals (APs, or program managers) of the nonacademic 
activities. 

Delivery of Activities 

The following section describes the delivery of activities, providing information on 
equipment used, population served, and activities implemented It details the common trends in 
program execution as well as the diversity that existed across sites. 

General Program 

The morning academic program used two computer software programs: ReadPro^ and 
MathPro^ (see Appendix B). All eight sites had both ReadProg and MathProg courses in 
place. Each site had at least one class that used each type of software; some had multiple 
classes. ReadProg and MathProg complement other, more traditional (noncomputer) courses 
and most children attended both the computer-oriented and more traditional classes. The 
frequency of ReadProg and MathProg classes varied from every day to twice a week. 

ReadProg sessions ranged from 45 minutes to VA hours, with the typical class being about an 
hour long. MathProg sessions tended to be a little longer, from a minimum of one hour to a 
maximum of five hours. The average MathProg session lasted about 1 14 hours (see Table 1).*^ 
The students who participated in these activities were in the 6th through 10th grades (Table 2). 



^ ReadProg is a fictitious name. We have not used the actual name of the program in order to avoid 
becoming involved in issues related to the marketing of proprietary software. 

^ MathProg is also a fictitious name. 

The tables are in Appendix A. 
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Setting 



The typical activity took place in an average -sized classroom. Computers were set up 
on desks or individual computer stations that were grouped together in small clusters. Most of 
the computers were PCs, although a few sites used more recently pxirchased laptops (Table 1). 

Student Participation 

Students gave many different reasons for attending the summer school program, 
predominantly mentioning poor academic performance (either low grades or low SAT- 9 
scores), parental requests, and having “nothing better to do.” These reasons seemed to explain 
why students were in the ReadProg program, which is mainly for remedial purposes. The 
MathProg program is an enrichment program, so students were required to meet testing 
guidelines before they could be admitted^ ^ Ongoing participation in both programs was 
contingent on good behavior, attendance, and appropriate performance. 

Description of the Activities 

In a typical session, students came in from their previous class and sat at one of the 
computers. The facilitator might give a brief introduction, but usually the students simply logged 
into their accounts and started working. The entire session was spent doing exercises, reading 
lessons, taking tests, and pursuing other activities dictated by the software. The students worked 
independently, speaking occasionally with the facilitator when a question arose. At the end of 



“ The tests that were used differed across sites; they included the SAT-9, the Iowa Test of Basic Skills, and 
pretests specifically for MathProg eligibility. 
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the session, the students logged out of their accounts and headed off to their next activity for the 
morning. 

Promising Components 

Overall, staff and students saw the ReadProg and MathProg programs in a very positive 
light. This section details the components of the program that appear to be most promising. 

Resources 

Generally, sites were equipped with sufficient resources, including dedicated staff and 
hi^“ quality hardware and software. 

Staff 

The quality of adult supervision was quite impressive. Classes tended to be very 
reasonably sized. The size of the ReadProg classes varied widely, from 3 to 20 students (Table 
1). The average number of students was about 14. The MathProg classes were somewhat 
smaller, averaging around 8 students, although the range was from 3 to 1 8. Many of the larger 
classes also had aides or classroom teachers, so the average student- to- facilitator ratio was 
around six to one (Table 2). 

Because the gpal of the technology component was to increase student familiarity with 
computer technology, it was interesting that a significant number of the facilitators lacked 
knowledge of common computer packages such as the Microsoft Office applications, and only 
a few considered themselves to be computer experts. One expressed a “reluctance to moving 
away from traditional classroom methods,’’ and two felt inadequately prepared in terms of 
technology training. Nevertheless, all the facilitators felt comfortable teaching the programs they 
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were using with the students, probably in large part because they had received training in those 
programs. Many of the instructors use the ReadProg software during the year and therefore had 
received previous training. Newer ReadProg and MathProg instructors, as well as some of the 
experienced ReadProg staff, had received training just before the program started. This training 
varied greatly in duration and activities, from daylong sessions to simply running the MathProg 
tutorial. In addition to technical training, most facilitators teach in DCPS during the school year 
and felt comfortable with the. academic material for which they were responsible. 

Facilitators appeared to be effective in both assisting their students with academics and 
monitoring their behavior. Most facilitators were observed circulating in the classrooms to check 
student progress and help students who had questions or concerns. Students seemed very 
comfortable asking facilitators for help and seemed satisfied with the assistance they received. A 
few facilitators seemed especially helpful, telling their students not to get frustrated, joking with 
them, patting them on the back, and giving positive feedback. 

All facilitators identified goals for their students, although these goals varied greatly in 
both topic and specificity. Most facilitators focused on the intermediate goal of program 
progression and the ultimate goal of skill improvement/increased knowledge. In terms of 
program progression, ReadProg facilitators had varying expectations, including advancement to 
a specific reading comprehension level and completion of the entire program. In MathProg, 
most of the facilitators had goals for advancement to higher levels of math 
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(prealgebra/algebra).*^ Nearly half of the ReadProg facilitators said they hoped for some 
improvement in reading level, (one specifically wanted a one -level increase, while another 
wanted a two-level increase); five of the seven MathProg facilitators wanted students to either 
gain more pertinent knowledge or enhance existing skills. Additional aspirations included SAT-9 
improvement, grade level advancement, improved computer literacy, and increased confidence. 

The majority of the facilitators said that they communicated progress to their students, 
most commonly through the use of progress charts posted in each classroom. Some of the 
facihtators also mentioned using e-mail and one-on-one conversation. Most of the facilitators 
also used a rewards system, although in some cases they appeared to be giving away gifts 
unconditionally rather than as an incentive for good behavior. One particularly generous teacher 
planned to give the student who made the most progress a graphing calculator at the end of the 
summer. Another facihtator commented that she instituted a reward system after seeing that ‘‘the 
kids needed more motivation than the stars on the wall.” Typical rewards included food, 
coupons, money, movie passes, and CDs. It seemed that most facilitators were paying for these 
material rewards themselves. Both the ReadProg and MathProg software generate certificates 
for mastering certain skills or topics. These printable certificates were used in many of the 
classrooms for additional positive reinforcement. Most facilitators had some way to measure the 
achievement of the goals they set. Seven of the sixteen said they used some sort of pre-/post- 
testing to measure progress. Most of the facilitators did not specify the nature of these tests, 
although one creative MathProg instructor said that she let the students take the final exam as a 

For some, it was unclear whether they were referring to advancement in school or advancement through 

9 




12 



pretest and used it again as a posttest at the end of the program. Over half of the facilitators said 
they used features of the software to measure progress. ReadProg facilitators used the progress 
charts and the “Results Room,” while MathProg instructors used the built-in tests. Other 
measures include SAT-9 scores, passing the summer school course, personal communication, 
and homework checks. 

Physical Resources 

The quality and quantity of equipment and facilities were exceflent. Each activity took 
place in a classroom that was clean, well- lit, and spacious. Most sites had air-conditioning, 
although three did not, and heat was a problem in at leaist one of these. In every observed 
activity, each child had access to his or her own computer, and most classrooms had many 
more computers than students. The average was eight computers per student, and the minimum 
was one per student (Table 1). The computers may be used at a higher rate during the regular 
school year. All but one of the classrooms had Internet access. 

All but one of the facilitators said they had access to skilled technical assistance, but 
satisfaction with the assistance varied by site. One facilitator complained about having problems 
with some of the computers that were not addressed for more than two weeks. On the other 
hand, another facilitator said that when the power went out one day, “it seemed like they were 
here in seconds.” Overall, it seemed that technical assistance was available and was satisfactory. 



the program. 

In this classroom, students were using a version of ReadProg that does not require Internet access. 
The remaining facilitator did not comment on this issue. 

It seemed that this facilitator was able to conduct the designated activities despite this issue. 

10 
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And even though at least some computers were not functioning at most sites, there were 
generally far more functioning computers than students in the classrooms we observed. 

Software 

On the whole, facilitators were very pleased with both ReadProg and MathProg. One 
satisfied MathProg facihtator noted, “It teaches almost everything I would have done by hand,” 
while another commented, “I’ve looked at many algebra books in my time, and this is great.” 
Some ReadProg facilitators were similarly pleased, one going so far as to call the program 
“phenomenal.” MathProg facilitators enjoyed the immediate feedback of the program, ease of 
checking progress, self-explanatory nature, extensive practice, good homework, and the fact 
that it requires a low level of supervision. ReadProg facilitators liked the ease of use, immediate 
feedback, options for checking progress and adjusting skill levels, pre- and posttests, and the 
fact that the program motivates and interests students. 

All but one of the facilitators were satisfied with the way the software accommodated 
students of different skill levels. The dissenting party felt that when students are at different skill 
levels, “that’s where the technology stops and the teacher has to step in.” However, most 
agreed with one ReadProg instructor, who remarked, “Students can go at their own pace, so 
we don’t have the problem of the advanced kids being held up by the slower learners.” Many 
facilitators of both programs especially liked the fact that the software was self-paced and that it 
was easy to determine when students encountered difficulties. Some MathProg facilitators were 
also pleased that students could test themselves on their own when they were ready. 

Both MathProg and ReadProg have impressive management capabihties that nearly all 
facihtators used in some way, either daily or every few days. The most common use was to 
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check progress or update the progress charts — more than half of the facilitators mentioned this 
capability. Two facilitators said diey used the ReadProg. capability for custornizing each 
student’s program, and one MathProg teacher said he spent one to two hours after every class 
checking up on the students. 

Student Behavior 

Discipline 

In general, students exhibited commendable behavior. FaciUtators kept the rooms quiet 
and orderly through active supervision. There were very few incidences of inappropriate 
behavior, and most were addressed quickly and effectively. There were no cases of students 
being removed from the activities or of any other major disciplinary action. The mild behavioral 
problems mainly involved students talking or getting out of their seats. 

Engagement 

Most students exhibited a high level of engagement in both ReadProg and MathProg 
sessions. Students were actively engaged throughout the sessions: working diligently on the 
matching exercises in ReadProg or reading notes and solving homework problems in MathProg. 
On average, 93 percent of the students in both ReadProg and MathProg classes were judged 
as being very or somewhat engaged in their activities. The level of engagement varied widely 
across the classes, but all students in one -quarter of the activities were judged very engaged 
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Problematic Components 



While the DC 2V^ CCLC program was successful in many ways, some problems were 
observed that suggest cause for concern. By bringing these issues to the attention of DCPS 
staff, we hope to generate a dialogue that may help to enhance the quality of program 
implementation. 

Student Involvement 

Student Interest 

Although the overall level of engagement was high, the level of interest and enjoyment of 
the activities was less impressive for some students. While students using ReadProg seemed to 
be generally engaged — in the sense that they were actively working — they did not seem to be 
especially attracted to the program. In some focus groups, students strongly disliked ReadProg, 
asserting that it was “stupid,” ‘l^oring,” and “pointless.” A facilitator remarked, “One negative 
that I found — some students, when they get to a certain point, I hear complaints that if s boring. 
It*s hard to keep them motivated when they get to that point.” Some students found the program 
frustrating, especially in its repetitive nature. Many of the students in one focus group 
complained that ReadProg takes too long, because the activities have to be repeated until 
students can perform them with a consistent speed and accuracy. A boy in this group said that 
he did one activity 25 times. Two of the facilitators agreed that the repetition made it hard for 
the students to stay motivated One facilitator further elaborated that this issue, as well as the 
excessive number of exercises, were program flaws. That same facilitator thought the students 
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should have more choice over the activities in order to curb boredom. Despite the disinterest 
and iBrustration of some individuals, many others liked the program and found it interesting. 

There was some dissatisfaction with MathProg as well. Some students wished the 
program had been more challenging and exciting. One if those who found it extremely boring 
said it felt like “math takes forever.” Some students suggested adding games to improve the 
program. One group said that they preferred another program. 

While some students were turned off by MathProg, others exhibited remarkable interest 
in the program. One facilitator explained this by saying, “This is an enrichment class. Kids 
choose to be here.” 

Motivation 

There was little evidence that most students were either excited about or motivated by 
their progress. Students generally did want their certificates of progress, and in a couple of 
cases students were observed checking their progress charts or showing some excitement about 
getting certificates. Overall, however, there was little evidence that students understood or cared 
about their progress. However, several facilitators strongly felt that the students were motivated 
by their own progress; one said, “They will remember their test scores. One of the students got 
100 percent. It’s nice to see the excitement.” 

Decrease in Engagement Over Time 

While the overall levels of engagement were impressive, in some of the classes we 
observed, engagement decreased during the class period Notably, in one MathProg classroom, 
all the students started out working well, but by the end of the 80-minute period, over half had 




14 



stopped working completely. A ReadProg facilitator said, “They start off well, but they get a 
little antsy.” 

At sites with unengaged students, few efforts were made to reengage them, and those 
that were made had little success. Such efforts were deemed unnecessary in many cases, as 
students were seldom doing more than taking a minute to daydream between activities before 
getting back to work. However, there were other, more serious cases in which students did not 
become reengaged. In one extreme example, a student stopped his work on MathProg to use 
the Internet. The facilitator asked the boy to return to his work, but when the boy resisted, the 
facilitator stopped trying, saying only, “This is the last time Pm going to let you do this.” 

Illicit Internet Use 

When students were not actively engaged, they frequently chose to surf the Web. One- 
third of the facilitators identified the forbidden use of the Internet as a concern. One particularly 
adamant facilitator detailed the problem: ‘We’re dealing with younger students who are easily 
distracted, and some of them have learned to minimize windows so they can be on the Internet, 
and then minimize the Internet window whenever I walk by.” This teacher’s recommendation for 
improving the program was a “closed shop” for the Internet, which would allow students access 
only to the MathProg site. The problems were confirmed in our observations: We observed 
illicit Internet use at nearly half of the sites. While there was some feeling among both students 
and facilitators that the level of Internet use was “discouraged, depending on the teacher,” it 
seemed difficult for many facilitators to act Most often, students were seen rninirnizing the 
Internet window when an adult walked by, a practice that many students admitted to in the 
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focus groups. In one class, students took advantage of an opportunity to surf the Web when the 



teacher was helping another student. 

In addition to the issue of general Internet use, some sites experienced further problems 
with the content of the sites being accessed Two facilitators learned of students accessing 
pornographic sites by reviewing the histories on their computers, and our own observations at a 
few sites corroborated this. Only four facilitators knew of the existence of any Internet filters at 
their sites, and two of them believed that the students could still access inappropriate material. 
The students did not report visiting any of these web sites but did mention playing games, 
listening to music, and looking at World Wrestiing Federation sites as common activities. One of 
the focus groups noted that there were some efforts to stop Internet use during activities, 
mentioning that using the Internet could result in expulsion from the program. This is also a 
problem that has received a great deal of attention nationally (National Research Council and 
Institute of Medicine, 2001). 

While illicit Internet use was a problem at many sites, half of the facilitators allowed their 
students some time on the Internet, usually 5 to 15 minutes at the end of the class. Most of this 
use was direcdy linked to either performance (completing a certain number of exercises) or 
effort (working well for the day), or simply as a break. 

Low Attendance and Enrollment 

Using computers to improve academic achievement is expected to increase student 

interest, motivation, and engagement One might also hope that enrollment in these programs 

would increase as students become aware of the technology options. Unfortunately, the 

enrollment and attendance numbers suggest that many Washington, D.C., youth are not taking 
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advantage of the opportunities available through the DC 2V^ CCLC program. For instance, as 
Table 3 shows, less than a quarter of the students enrolled during the school year in these 
schools attended the summer program in 2001 (The range is 13 percent to 41 percent.)^^ 
Indeed, the total number attending (920) was lower than the number for the summer of 2000, 
when almost 1,000 students attended (Raphael 2000b), even though there was one more site 
open in 2001.*^ 

Enrollment and attendance issues impeded implementation, with 6 of the 16 activities 
reporting such difficulties. Notification that one of the schools would open for the summer was 
made quite late. Enrollment at that school went from 85 the first week to 160 the second 
week.*^ In one of the MathProg classes, testing for eligibility did not begin until after the 
program started. Students who had been recommended by teachers were sent to the facilitator 
over the course of two weeks and did not begin working on MathProg or ReadProg until after 
they had been tested There were also a variety of scheduling problems, including missing 
rosters and lack of coordination with other summer school activities. In addition to hampering 
the initial implementation of the program, some problems were not resolved weeks later. On 
one site visit, the six students for the facilitator’s first class never arrived, and the facilitator did 
not know why. Other attendance issues were experienced as well. There was some sporadic 



These are approximate estimates, as some students attended programs at other schools. 

This comparison is complicated by the fact that some primary school students were included in these 
attendance numbers, and the number of primary schools with their own programs may have changed during 
this period. Further discussions of enrollment are included in Appendix B of the companion report, Russell 
etal(2002). 

We were also told that all schools experienced a large increase in enrollment during the first week, because 
many students had not been told that they had to attend summer school, which was mandatory for those 
who scored poorly on the SAT-9 (the standardized test given to all DCPS students). 
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attendance resulting from hot weather, and some students who had to walk to school were late. 
Two teachers said that attendance dropped over time as students started summer jobs, camps, 
and vacations. Another facilitator, who had been given very short notice about her position, was 
sure that the enrollment would have been higher if there had been more advertising for the 
program. Many facilitators commented on the extent to which attendance affected success. A 
ReadProg facilitator said, "Those who come on a regular basis are on schedule, but those who 
are not coming regularly are not achieving the goals set forth for them.” 

Other Issues 

Lack of Individualized Treatment 

A major benefit of using these software packages, and one that many facilitators 
mentioned, was that they could customize the program for each student. However, for both 
ReadProg and MathProg, everyone seemed to begin at the first level, even though students 
spanned a four-year grade range and in spite of the fact that pretests were given to all the 
ReadProg students. The facilitators’ responses to starting all students at the initial level were 
mixed. Some thought that it was easier to see progress and identify appropriate skill levels if the 
students worked up to where they started to struggle. Others thought that students should be 
placed in different skill levels by pretesting. 

Starting everyone at the beginning of a program could affect both student interest and 

their gains in knowledge and skills. Particularly striking is the fact that MathProg is supposed to 

be an enrichment program, yet students were starting from basic math. Thus, many students and 

facilitators said that the program was mainly a review of previous material. One facilitator had 

students who ‘Svent straight to the tests — no lessons — passed and moved on,” which indicates 
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that these students already knew the material. Some students said they found the MathProg 
program boring and wished it were more challenging. 

A common concern with ReadProg was that it is too basic for middle school students, 
which can make the activities both ineffective and boring. One facilitator said, “There’s 
something that needs to be added to it for students who don’t need remediation. For elementary 
students it would suffice, but for middle school students something else needs to be added that’s 
a little more challenging.” Many of the students also felt that ReadProg was too easy, some 
speaking with great disdain of the ‘haby words” they were given. It is unclear whether these 
problems are attributable to having all students start at the beginning, because the more able 
students should have been able to progress through the basic materials fairly quickly. 

Another problem with ReadProg is that students do not get to reading comprehension 
activities until they have they have cornpleted many exercises dealing with subskills and 
phonemics. Judging from observation, very few students make it to the reading comprehension 
activities, and nearly half of the facilitators wished that their students had been able to spend 
more time working through those activities. In fact, two facilitators said that students should be 
filtered into the reading comprehension activities. Some facilitators identified this as a software 
flaw, but our observations suggest that facilitators could have started the students at different 
levels. On the other hand, it is possible that students would feel they were being treated unfairly 
if they were started at different levels. 

Lack of Support from Aides 

While the quality of the facilitators was generally excellent, there were some problems 

with aides. In nine of the activities (eight were ReadProg programs), additional adults were 
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present. These other adults were mostly classroom teachers or hall monitors who work during 
the school year. Also, two classes had interns from George Washington University as aides. In 
nearly half of these classes, these other adults were very effective in their roles, performing tasks 
similar to those of the main facilitator. In the remaining classes, however, they seemed to have 
little interaction with the students and were mostly occupied using the computers and looking at 
papers. Some facilitators believed that they would have benefited from more assistance from 
these aides. A ReadProg facilitator spoke about one effective classroom teacher: “He wanted to 
know where they need help, so that in his classroom he can help with that. He knows the 
computer program and was really eager to learn.” But this facilitator noted that “not every 
facilitator and teacher have this same relationship.” 

The lack of assistance from classroom teachers may be partially explained by their 
unfamiharity with computer activities. One facilitator commented on the lack of technical skills in 
many of the classroom teachers: “One problem is that not all adults are computer literate. Some 
older teachers are scared and do not want to work with the computer in front of the kids, 
because some children may see this as a deficiency.” Another facilitator agreed, remaridng that 
if classroom teachers are interested and make an effort to learn the technology, they can be of 
enormous help. 

MathProg Software 

Although the MathProg software was generally well received, a few concerns surfaced 

in the observations and facilitator interviews. The first involves the online tests. In the future, the 

tests will change, but currently they remain exactly the same each time a student takes them. In 

addition, they use multiple choice answers. In a few classes, the chapter tests were used as 
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pretests, so the students could take them as many times as they wanted. The facilitators believed 
that this approach allowed students to move at their own pace and avoid review material, but it 
may also have allowed them to guess at answers xintil they got them correct instead of working 
out the answers. On one of the site visits, students taking one of the tests were observed writing 
down answers and performing very few calculations when they retook the tests. The facilitator 
said that the students wrote down the answers so they would not lose the information if the 
computer crashed while they were trying to input them. However, when one student taking the 
test was questioned, he did not know the meaning of the term “pi,” nor did he know the 
formulas for area and mean, all of which were used in questions he had already answered 
correctly. Also, a few students complained that they had to take the test too many times before 
they passed it, perhaps indicating that students were not studying enough before the tests. 

Another possible problem with MathProg is that the answers to the homework 
problems are easily available. They can be viewed by simply scrolling down farther on the same 
screen as the questions. Although one facilitator liked the fact that the program provided 
immediate feedback about the problems, he also believed that this might encourage students to 
cheat. This problem did not appear to be widespread, as only one student was observed 
copying the answers directly onto his answer sheet without performing any calculations. 

One facilitator deemed MathProg appropriate for younger children but below the skill 
levels of many of the students in the program. 
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Other Observations 



Though not necessarily harmful or helpful to the program, our study yielded a few 
additional notable observations. 

Computer Use Outside of Summer School 

Questioned in the focus groups, the students seemed to have very little interest in 
learning new computer activities in the program. One group said that if they wanted to learn 
things like games or e-mail they could do it on the weekend Some students mentioned learning 
typing (one student said that they don’t type, they “pluck”). Only a few students mentioned 
wanting to learn other activities, including playing games, making shapes as part of a math 
program, downloading pictures, and creating web sites. 

This disinterest may be attributable to the fact that most of the students already seemed 
very comfortable with computers and used them outside of school. An impressive 27 of 33 
students asked had access to a computer at home. Those who did not used them at the library, 
church, and their friends’ houses.*^ Many seemed familiar with e-mail and the Internet. Twenty- 
two of 29 students asked had their own e-mail accounts. A few in each group were familiar with 
PowerPoint and Excel, and a handful maintained their own Web pages. The frequency of 
computer use varied but was generally high. Some students said they used computers only 15 
minutes a day, but many others said they did so “all the time” or “whenever I get a chance.” 
These patterns are somewhat surprising, given recent evidence of a large “digital divide” by race 
in access to computers at home. For instance, Newburger (2001) reports that far fewer than 
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half of black and Hispanic children ages 3 through 17 lived in households with computers in 
August 2000, and fewer than 20 percent of these children used the Internet at home.^® Since 
over 90 percent of the students enrolled in the schools we observed were either black or 
Hispanic, their self-reported computer xise suggests that these students may have far more 
access to computers at home than the average student in these schools. 

During the school year, there is variation across schools in terms of academic computer 
use. Some of the students had used ReadProg during the school year, and students at one site 
said they had used MathProg. Others said that they had used other academic software 
packages during the school year. Students at one site specifically mentioned Academy of Math. 

Facilitator Suggestions 

In addition to commenting on existing features of the program, facilitators offered a wide 
range of suggestions for additions and improvements. Some of the suggestions for MathProg 
were to add an auditory component, allow students to do problems on the computers (instead 
of on paper), and provide better examples. ReadProg facilitators recommended a time limit be 
added for finishing each section, a better reward structure be implemented, and the program be 
put on disk so that students could woric at home. 



Interestingly, none mentioned community centers, though such centers are available in Wards 6, 7, and 8 
(Manjarrez et al, 2002). 

In comparison, more than two -thirds of white non-Hi spanic and Asian children were in homes with 
computers, and more than one-third used the Internet at home. 

23 



O 

ERIC 



26 



Conclusion 



During the summer of 2001, the DC 21^^ Century Community Learning Center program 
helped provide computers and software designed to improve the academic success of summer 
school students from 10 junior high and middle schools in Washington, D.C. In addition to 
helping to reduce the number of youth left idle during the summer, these activities appeared to 
have the potential to improve student familiarity with computer technology. This last point is 
important, because “technofluency” is an important skill for the modem labor market (Krueger, 
1993). 

Our observations suggest that the quahty and quantity of equipment were high, as was 
the overall level of exposure to technology. The staff seemed well prepared and dedicated, and 
the students were generally engaged Although the overall program was impressive, challenges 
remain. Low student enrollment and illicit use of the Internet were two impediments to 
successful student outcomes that we observed. However, the optimistic and determined attitude 
of both facihtators and management should help to ensure that program implementation 
improves over time. 

Student outcomes could not be directly measured from the data we collected. 

Therefore, for future DC 2L* CCLC programs, we recommend that a carefully designed impact 
evaluation plan be used We also strongly suggest continued self- monitoring by DCPS staff. 
These measures should help improve the program for the benefit of future Washington, D.C., 
youth. 
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Appendix A: Tables 



Table 1: Equipment for DC 21st CCLC AM Academic Technology Components, Summer 2001 


Mathematics 


School 


No. of 
Computers 


No. of 
Students 


Computers 
per Student 


Computer Type 


Internet 

(Y/N) 


Duration of 
Session 


Eliot JHS 


24 


16 


1.5 


PC desktop 


yes 


1 hr 


Francis JHS 


48 


7 


6.9 


PC desktop 


yes 


1.5 hrs 


Garnet-Patterson MS/Shaw JHS 


30 


8 


3.8 


PC desktop 


yes 


1-1.5 hrs 


Kramer MS 


10 


3 


3.3 


PC desktop 


yes 


55 min 


MacFarland 


24 


6 


4.0 


PC desktop 


yes 


5 hrs 


Sousa MS 


16 


9 


1.8 


PC desktop 


yes 


2 hrs 


Sousa MS 


16 


12 


1.3 


PC desktop 


yes 


2 hrs 


Terrell JHS 


28 


7 


4.0 


Mac, PC desktop 


yes 


1 hr 20 min 


TOTAL for MATH 


196 


68 


N/A 


N/A 


N/A 


N/A 


Minimum 


10 


3 


1.3 


N/A 


N/A 


55 min 


Maximum 


48 


16 


6.9 


N/A‘ 


N/A 


5 hrs 


Average 


25 


9 


3.3 


N/A 


yes (mode) 


1 hr 42 min 



* No math class was observed for Harris/Hart because the facilitator was gone for the entire week 



Reading 


School 


No. of 
Computers 


No. of 
Students 


Computers 
per Student 


Computer Type 


Internet 

J/N) 


Duration of 
Session 


Eliot JHS 


24 


14 


1.7 


PC desktop 


yes 


1 hr 


Francis JHS 


48 


20 


2.4 


PC desktop 


yes 


1.5 hrs 


Garnet-Patterson MS/Shaw JHS 


25 


12 


2.1 


PC desktop 


yes 


45 min 


Harris Educational Center/Hart MS 


30 


13 


2.3 


PC desktop 


yes 


1 hr 15 min 


Harris Educational Center/Hart MS 


30 


16 


1.9 


PC desktop 


yes 


55 min 


Kramer MS 


50 


14 


3.6 


PC desktop 


yes 


55 min 


MacFarland 


24 


17 


1.4 


PC desktop 


yes 


1 hr 


Sousa MS 


20 


20 


1.0 


PC wireless laptop 


yes 


1 hr 


Terrell JHS 


25 


3 


8.3 


24 laptop, 1 desktop 


no 


45 min 


TOTAL for READING 


252 


115 


N/A 


N/A 


N/A 


N/A 


Minimum 


20 


3 


1.0 


JM/A 


N/A 


45 min 


Maximum 


50 


20 


8.3 


N/A 


N/A 


1.5 hrs 


Average 


32 


14 


2.9 


N/A 


yes (mode) 


1 hr 



Source: Observations by Urban Institute staff 
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Table 2: Characteristics of DC 21st CCLC AM Academic Technoloqv Programs, Summer 2001 


Mathematics 


School 


No. of 


No. of 


No. of 


Age/Grade 


Students/ 


Facilitators 


Aides 


Students 


of Students 


Facilitator 


Eliot JHS 


1 


0 


16 


grades 6-8 


16.0 


Francis JHS 


2 


0 


7 


grades 8-9 


3.5 


Garnet-Patterson MS/Shaw JHS 


1 


0 


8 


grades 6-8 


8.0 


Kramer MS 


1 


0 


3 


grades 6, 8 


3.0 


MacFarland 


1 


0 


6 


grades 6-7 


6.0 


Sousa MS 


1 


0 


9 


grade 7 


9.0 


Sousa MS 


1 


0 


12 


grade 7 


12.0 


Terrell JHS 


1 


0 


7 


grades 6-8 


7.0 


Total 


9 


0 


68 


N/A 


N/A 


Minumum 


1 


0 


3 


grade 6 


3.0 


Maximum 


2 


0 


16 


grade 9 


16.0 


Average 


1.1 


0.0 


8.5 


grade 7 


8.1 



*No Math class was observed in Harris/Hart because the facilitator was gone for the entire week. 



Reading 


School 


No. of 
Facilitators 


No. of 
Aides 


No. of 
Students 


Age/Grade 
of Students 


Students/ 

Facilitator 


Eliot JHS 


1 


2 


14 


older MS 


14.0 


Francis JHS 


2 


2 


20 


MS 


10.0 


Garnet-Patterson MS/Shaw JHS 


2 


0 


12 


MS 


6.0 


Harris Educational Center/Hart MS 


1 


2 


13 


age 13-14 


13.0 


Harris Educational Center/Hart MS 


1 


1 


16 


grade 6 


16.0 


Kramer MS 


1 


0 


14 


grade 7 


14.0 


MacFarland 


1 


1 


17 


grades 6-8 


17.0 


Sousa MS 


1 


2 


20 


grades 6-8 


20.0 


Terrell JHS 


1 


0 


3 


grades 8-10 


3.0 


Total 

Minumum 

Maximum 

Average 


11 

1 

2 

1.2 


10 

0 

2 

1.1 


129 

3 

20 

14.3 


N/A 

grade 6 
grade 10 
grade 7 


N/A 

3.0 

20.0 
12.6 



Source: Observations by Urban Institute staff 
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Table 3: Students in DC 21st CCLC AM Programs, Summer 2001 


School 


AM 

Enrollment 

According 

to 

Monitoring 

Reports 


AM Attendance 


Ratio of 
summer 
attendance 
to school 
year 

enrollment 


Observed by 
Ul staff 


Monitoring 

Reports 


Eliot JHS 


79 


38 


56-60 


0.156 


Francis JHS 


61-105 


58 


49-64 


0.128 


Gamet-Patterson MS/Shaw JHS 


161-229 


130 


132-182 


0.181 


Harris Educational Center/Hart MS 


309 


221 


148-179 


0.137 


Kramer MS 


62-63 


60 


32-45 


0.163 


MacFarland 


298-375 


221 


155-302 


0.415 


Sousa MS 


181 


125 


111-132 


0.319 


Terrell JHS 


107-164 


67 


93-149 


0.386 


TOTAL 


1258-1505 


920 


776-1113 


N/A 


Minumum 


61 


38 


32-45 


0.130 


Maximum 


375 


221 


155-302 


0.410 


Average 


190 


115 


118 


0.235 



Sources: DCPS Monitoring reports were written every few days by staff hired by the director of 
the DC 21^ CCLC program. Ranges reflect how schools’ student enrollment changed over the 
summer. Different methods of collection used account for most of the differences across 
columns. The Urban Institute’s numbers are based on one -day visits to the various schools. The 
last column uses total student enrollment, as found at http://www.dcschoolsearch.com. 
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Appendix B: ReadProg and MathProg Software 

ReadProg 

ReadProg is a software program intended for reading improvement; it is generally 
designated for remedial purposes. It focuses on teaching the “component” skills of reading 
(identification of individual sounds and syllables), in hopes of improving overall fluency. All 
activities are accompanied by tutorials and practice sessions to guide the student. But while 
ReadProg allows students to work independently, it requires significant assistance from an 
instmctor to reach its fullest potential. 

ReadProg is an interactive program that leads the student through a progression of short 
lessons that use speech, sound, graphics, and animation. Each lesson involves one of three main 
activities: word matching, sound matching, or reading comprehension. Word and sound 
matching are similar activities — the object is to match a target sound or group of letters to one 
of three choices. In the word matching activity, the target word appears on the screen and the 
three choices are displayed below it. The student uses the number keys to select a response. In 
sound matching, the computer speaks the target word and the matching options, then the 
student uses the mouse both to repeat sounds and to make a final choice. In both matching 
exercises, the program immediately tells the student whether the answer is correct or not, then 
moves on to a new group of words or sounds. The students must repeat the matching exercise a 
certain number of times, then they are given their accuracy and speed results. Students may 
proceed only after they have completed three exercises at similar speeds and at a target level of 
accuracy. As students master these exercises, they begin to get more complex. The progression 
starts from single letters or soimds and moves to complete words. The exercises vary among 
matching the exact word, the beginning or ending sounds, and rhyming words. 

The reading comprehension component encompasses 10 levels of exercises in oral and 
silent reading. It is designed to improve students’ abilities to identify the main idea, understand 
relationships, make inferences, note specific facts and details, and retain information. In these 
exercises, the students have unlimited time to read a short selection, then they must answer five 
mult^le- choice questions about what they Just read. If any of the questions are answered 
incorrectly, the student must respond to them again, but this time they are able to view the 
paragraph. 

ReadProg has a variety of built-in capabilities to help both the instmctor and the student 
with the program. It offers tools for customizing the individual student’s program, including 
changing the number of repetitions in a given activity and altering the skill level in the reading 
comprehension exercises. A function called “Teacher Time” is activated whenever a student is 
having too much difficulty with an exercise: It fioezes the student out of activities until the teacher 
attends to the problems. Also, for every skill mastered, a virtual award appears in the “Results 
Room,” and the student receives a printable certificate. Additional features include diagnostic 
tests, automated interpretation and report writing for these tests, a message communication 
system, and an online library. 
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MathProg 

MathProg is an online math training program. Once a teacher registers a class of 
students, those students can log on individually from any terminal connected to the Internet. 
Students can work independently through much of the program, but teachers are an integral 
complement to the technology, monitoring progress and helping students through problem areas. 

MathProg provides training in five courses: Basic Math/Pre- Algebra, Elementary 
Algebra/Algebra I, Intermediate Algebra/Algebra II, Plane Geometry, and Test Prep Course. 
Each course is broken down into chapters, which are further divided into lessons. Each lesson 
follows the same basic format; they are composed of text on the topic interspersed with 
animated and interactive examples. Students can read through lessons at their own pace and 
refer back to them later as well. Each lesson also contains many relevant homework problems; 
students do these problems on paper and compare their answers with the answers online. To 
proceed to the next chapter, the student must take an online multiple-choice test, which is 
scored immediately by the computer. The teacher sets the computer in advance to determine the 
percentage correct that will allow the student to move on. 

Like ReadProg, MathProg includes a number of management capabilities. Through a 
single terminal, instmctors can access records for all their students, including test scores, grades, 
and lesson times. They can commimicate with students through a built-in message system. The 
program also allows for some customization, letting facilitators choose the order in which 
students will progress through the lessons,^ ^ give students access to solutions, and determine the 
passing percentage. 



In the “closed” format, users must follow a standard progression of lessons; in the “open” format, they 
can skip around as they wish. 
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Appendix C: Protocols for Site Visits 



Session Observation Guide 



School: 

Activity/Facilitator: 

Date: 

Preliminary Instructions: Please make sure to introduce yourself to the facilitator. Allow 
him or her to introduce you to the group if it seems best, but try to remain as unobtrusive as 
possible. Please make sure to answer all the specified questions. If you have time and it seems 
feasible, you may informally question the students about their activities (suggested questions 
follow survey). 

Time session starts: 

Duration of session: 

Time observation starts: 

Time observation ends: 

Number of boys: 

Number of girls: 

Age/grade range of students: 

Number of facilitators: 

Number of aides: 

Number of computers: 

Type of computers used (list type and number: PC or Mac, laptop or desktop, wired or 
wireless, connected to Internet): 

Type of software being used (MathProg^eadProg/word 
processors/spreadsheets/Intemet/e-mail/other): 

Initial Information : 

1 . Describe the setting (room, building, nonschool location, how computers are grouped 
etc.) for the activities. 

2. If you were present at the beginning of the session, how were expectations for the 
session communicated by the facilitator, if at all? What did you think was the “plan” for the 
session? 

3. Did students seem to know what was expected of them (e.g., did they quickly 
engage in the activity, ask questions, appear interested)? 

Monitoring of Activities 
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Please note that specific activities will likely change over the course of the session; 
therefore, it is important to keep a running log. Use the questions below as a guide for what to 
report. As the specific activities change, please continue to update your responses. 

4. Describe the activities , providing details about what materials and manipulatives are 
used, what the actual activities are, how the activity is being accomplished, time spent explaining 
the activities, time on each activity, time spent actually using computer, etc. 

5. What are the students doing (e.g., practicing typing, surfing the Internet, downloading 

MP3s)? 

6. How are students organized (whole-group, small groups, pairs, working 
independently)? 

7. How are students interacting with one another ? Be sure to note examples of 
collaboration, cooperation, praise, feedback, and shared responsibility for activity. 

8. How is the facilitator interacting with students ? How are aides interacting with 
students ? Be sure to note facilitation, feedback, co-leaming, guiding, and demonstrations. 

9. Approximately how many students are engaged in each of the activities at hand? 
What are the unengaged students doing? 

Summary : 

10. Describe the noise level. Does it appear to be a problem? 

11. How is time spent during the observed period? 

Percentage of time working independendy: 

Percentage of time working in small groups: 

Percentage of time working in pairs: 

Percentage of time entire group spends listening to facilitator 

Percentage of time that students can choose their activities (free time): 

Other ( ): 

12. Do students have the opportunity to take responsibility for any aspects of the 
activity (e.g., may they choose which programs to use, what information they find on the Web)? 
Do they do so? 

13. What did you perceive as benefits of this activity for students? Please explain, using 
examples, quotes, description, etc. 

14. How interested are students in doing this activity? Are they aware of and motivated 
by their progress? Please provide examples and support for your assessment. 
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15. Would you describe this activity as structured, semistructured, or unstructured? 



Why? 



16. How engaged are students in this activity overall? 

Percentage of students very engaged: 

Percentage of students somewhat engaged: 

Percentage of students not engaged: 

17. What are students who are somewhat or not engaged doing (e.g., talking to each 
other, daydreaming, walking around the room)? 

18. Does the facilitator make an effort to reengage students who are not engaged? Do 
other students make such and effort? How? 

19. How would you describe the degree of supervision (e.g., from students actively 
supervised to neglected)? 

20. How is discipline maintained? How effective was the facilitator at managing the 
students? Please support your assessment. 

21. Were any students told to leave the room/area during this activity because of a 
discipline problem? Explain. 

22. Is the space adequate, in your opinion, for this activity? If not, why not? 

23. Are there adequate materials for all students (e.g., does every student have a 
computer, is the same software installed on all of the computers)? Explain. 

24. Was the community involved in ai^ way with this activity? Explain. 

25. If food was served, please answer the following: 

What food was served? How much? 

Did it appear fresh? 

Did it appear healthy? 

26. If parents were involved, please answer the following: 

How many parents were involved? 

What were the parents doing? 

Did the parents seem interested in the activity? 

Were they interacting with their own children? 

Were they interacting with other children? 
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27. Describe any interesting/relevant occurrences during this activity. Did you observe 
any trends in use based on gender, age, ability, etc.? 

Informal Questioning : If there is time and it seems appropriate, please feel free to ask 
the students informal questions. The following questions are suggested 

What are you doing? Have you ever used this program before? Do you ever use a 
computer outside of school? Do you like what you’re doing? 

Record any particularly interesting/relevant statements made by students or 

staff: 

Remember to thank the facilitator when you are finished. 
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Facilitator Interview Protocol 



If this is not a good time to talk, when can we call you to discuss this activity? 
Telephone number: 

My name is , and this is . We work at the Urban Institute and are 

part of the evaluation team for the DC 21st Century Community Learning Center program. We 

are very happy to meet you and to visit [school name] as part of our 

evaluation. We want to thank you for taking the time to meet with us and learn more about how 
the program is being implemented at the school. 

The purpose of our interview with you today, which should last about 30 minutes, is to 
understand the ways in which this program is working, as well as what isn’t working as well as 
anticipated and any lessons you’ve learned. We will be visiting each of the program’s DC sites 
and asking the same questions of program coordinators and students. We are not using this 
information to judge your work or to judge the results of the program. At this early stage 
in the program, we’re collecting information to be used to better understand program results. 

Because we are pursuing information that can benefit the future implementation of the 
program, we hope that you will feel comfortable talking candidly to us. All the information you 
provide will be kept anonymous. Any details about your program will be reported using phrases 
such as “in one program” or “one program coordinator explained that. ...” 

We would like to request your permission to tape the interview. These tapes will be 
kept confidential; we would just like to have them to check in case we miss anything, and 
possibly for training purposes. Taping the interview will help us focus better on what you are 
saying during the interview. 

Do we have your permission to tape the interview? 

Do you have any questions for us before we begin? 

Today’s Activity 

1 . How well do you feel the activity went today? [Observer to ask specific questions 
based on observation of period^ if appropriate.] 

Activity in General 

2. What are your program goals for students? [If they only mention improved test 
scores:] Are there any other goals? 

3. How are these goals measured? 
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4. How well are your students doing on achieving these goals? 

5. Are you doing any other activities to improve [reading comprehension/math 
skills]? 

6. If so^ how are these activities coordinated with the [ReadProg/MathProg] 
activities? 

I. Have you been able to implement this activity as planned throughout the summer? 
Please explain. 

Probes: Are students participating fully? Are some days or periods more difficult than 
others? What difficulties or challenges have you encountered in implementing this activity? 

Professional Development/Training 

8. Did you feel adequately prepared for or supported in your role as facilitator of this 
activity? 

9. Were skilled technology coordinators available to you when you needed them? 

10. Can you describe any training (or professional development) focusing on 
technology that you received to support your role in this program? 

What types of software were covered? Was the integration of this software into the program 

covered? How much training have you received? 

I I . What technology skills do you possess? 

Technology Issues 

12. What is your impression of the [ReadProg/MathProg] software program? 

13. Have you used the program *s Web~based management capabilities? What 
capabilities do you use? How often do you use them? 

14. Do you communicate with the students regarding their progress in the software 
programs? Do you alert the students to their progress? Do you reward them for 
successes and motivate them to advance? 

15. Do the software programs help you deal with students of different skill levels? If 
so, how? 

16. Are the students ever given the opportunity to access the Internet while 
participating in this activity? [By this we mean websites not related to MathProg 
and ReadProg. ] 

1 7. If so, are you aware of any Internet filtering technology in place that prevents 
students from viewing inappropriate sites? Is the technology adequate? Are you 
aware of any problems with this technology, or do you have any concerns about it? 
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General 

1 8. What suggestions do you have for improving this activity in the future? 
Thank the facilitators for participating. 
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Student Focus Group Protocol 



Introduction 

My name is , and this is . We work at the Urban Institute tere in 

Washington. We are conducting an evaluation of the DC 21st Century Community Learning 
Center program. We want to thank you for agreeing to meet with us. We are very happy to be 
at [name of school] to talk with you about this summer school program. 

We are visiting each of the 10 middle and junior high schools in Washington that have a 
summer school program like this one. We’re speaking to program coordinators and to students. 
Our discussion with you will last about 30 minutes. 

The reason we wanted to talk to you is that you, the students, are what this program is 
all about. We would like to learn how you feel about this program — ^what you’re getting out of 
it, what you think is working best, and what isn’t working as well. We would also like to ask 
you about what you think you’re learning in the program, and what you would be doing if you 
weren’t here. That sort of thing. 

We won’t use the information you give us to ‘‘grade” the program or to say whether it’s 
good or not, but to understand it better. 

We will keep the information you give us anonymous. That means no one, including 
your teachers and the program coordinator, will know what a particular student says during this 
discussion. Instead, we will report that “Students at one school said...” or “One female student 
felt that....” Because we’re keeping your responses anonymous, we hope you will tell us 
honestly how you feel. Your thoughts about the program will be very helpful to everyone 
involved We ask that you not talk about what anyone said during this discussion after we are 
done, so that everyone feels comfortable enough to say what they reaUy feel. Also, this 
discussion is voluntary — you do not have to respond to any question. 

We would also like to request your permission to tape our discussion. We’re the only 
ones who will use these tapes, to catch anything we missed the first time around Taping the 
interview will help us focus better on what you are saying right now, but again, no one else will 
listen to the tapes. 

Do we have your permission to tape this discussion? 



Do you have any questions for us before we begin? 
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1. You are doing a lot of different things in this program. Can you tell us about 
some of them? 

Probe: So the program is divided into different parts: , , 

. Can you talk a little about these parts? 



2. You are also spending some time on computers in this program. Can you tell 
us about what kinds of things you do on computers? 

Probe: So you spend time on MathProg, ReadProg, other learning activities, and 
recreational activities on computers. Can you tell us how much time you spend doing each? 



Probe: When do you get to do recreational activities on computers? Are there 
scheduled times for these activities, or are they only available after you have finished 
your work? 

[If only use computers after other work] Probe: How long does it take you to 
finish your work before you get to play on the computers? 



3. What kinds of things would you like to learn how to do with computers? 

Probe: Would you like to improve your math or reading skills with the computers? 
Would you like to learn how to type? Would you like to learn how to search the Internet or 
e-mail fiiends? Would you like to learn to play computer games? 

Probe: What do you already know how to do? Do you search the Internet about 
interesting topics? Do you e-mail fiiends? 

Probe: How many of you know how to use a word processor? 

Probe: How many of you know how to use a spreadsheet application? 

4. Do you use computers outside of school? Do you use the Internet outside of 
school? Where do you use computers outside of school? Can you use computers at 
home? 



Probe: How often do you use computers outside of school? How often do you use 
the Internet outside of school? What do you do when you use computers outside of school? 
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5. What kinds of things do you do with other students in the program? What 
kinds of things do you work on together on the computer? 

Probe: Do you work together on projects? What else do you do? [Ask for 
examples.] 

6. What kinds of computer activities do you do with your teachers in this 
program? 



Probe: Do they work individually with you on computers? What else do you do? 

7. Do you do computer activities in this program that you do not do during the 
regular school year? [Get examples.] 

8. What caused you to be in this program? 

Probe: Did you choose to get involved? Why? What had you heard about the 

program? Did your mother or father or someone else decide for you? 

9. Do you feel safe while you’re in this program? Do you feel safe in the school 
building? Do you feel safe getting home? 

10. What do you think you are learning, or getting out of this program? Can you 
give examples of what you mean? 

Probe: Have you learned anything new? Do you feel different? Do you do anything 
differently, or not do something you used to do? 

11. Are there any rules that affect whether or not you can be in this program? 

Probe: Do your regular school teachers, or anyone else, decide whether you can be 
in this program? 

12. What do you do in the summer when this program is not available? 

Probe: If you weren’t in this program, what would you be doing in the summer? 

13. Do you have ideas about how this program could be improved? Can you 
explain why you feel this way? 



Thank the students for participating. 
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